Computer mice, joysticks, and other manual tracking devices are ubiquitous tools for specifying positional information during human-machine interactions. With the advent of wearable computing, such bulky and obtrusive devices that, for example, generally require stationary surfaces for proper operation, are incompatible with the portable nature of apparatus that are designed to be worn on the body.
Wearable display devices include virtual reality (VR) displays such as those manufactured by Sony, Samsung, Oculus, Carl Zeiss; head mounted displays (HMDs) such as those produced by Google (e.g., Glass®) and Vuzix; augmented reality (AR) displays such as those manufactured by Microsoft, Vuzix, and DigiLens; and similar devices. Eye tracking can be used to view such displays and to specify positional information. However, the eyes are also used extensively during normal human activities.
Thus, a challenge when using eye position as an input data stream for interaction and control is to discern the intent of a user (DIU) based on eye movements. One of the goals of the systems and methods herein is to distinguish between movements of the eye that are associated with normal daily activities versus conscious or voluntary movements, herein referred to as “eye signals,” that are intended to interact with, and control, a smart device. A smart device is an electronic device, generally connected to other devices or networks via different wireless protocols such as Bluetooth, NFC, Wi-Fi, 3G, etc., that can operate to some extent interactively and autonomously.
Eye signal tracking can be considered to be a component within the field of machine vision that enables humans to communicate with machines. Eye signals are distinct from “gesture” controls since they must be discerned from the continual movements of the eye that are associated with daily living. In addition, eye signals can be affected by the user's environment (e.g., sunlight versus darkness), distractions, fatigue, mental state, cognitive load, sickness, drugs, and so on.
Eye signals can be used to either replace or supplement gesture-based HMI. Currently, the most common form of gesture controls involves tracking the face, hand, or one or more fingers. Other examples of HMI gesture control involve the monitoring of both hands, gait, overall body movements, legs, bracelets, fingers, rings, pens, and the like. The term “gesture control” has also been applied to interpret movements directed at touch screens, tablets, and other motion- or touch-sensing surfaces. In almost all cases when using these devices, gestures can be made that are under voluntary (i.e., conscious) control and that are distinct from normal daily activities.
Eye signal tracking using one or both eyes involves specific neural pathways and motor controls. Six muscles (superior rectus, inferior rectus, lateral rectus, medial rectus, superior oblique, and inferior oblique) control all forms of eye movement. In addition, the levator palpebrae superioris muscle controls movement of the eyelid. These muscles are innervated by three nerves (oculomotor, abducens, and trochlear) with nuclei in the brain stem. The structure of the eye including muscle attachment points coupled with the neural control of eye movements place anatomical and physiological bounds (e.g., range of motion in horizontal, vertical, and rotational axes; maximum velocities in each dimension; ability to remain stationary; movement precision; etc.) on eye movements.
Eye movements are classified as elevation, incyclotorsion, adduction, depression, extorsion, intorsion, and/or abduction. Unless blind, the eyes are considered an essential sense to conduct normal daily activities. Thus, algorithms that interpret eye signals must discern the intent of a user during eye movements (i.e., whether a movement is a part of an eye signal or serving some other function).
In addition, it is crucial to take into account the cognitive processing that is specific to eye movements. It is relatively easy, for example, for most individuals to generate circular motions with a hand without any cues or guidance. This is in sharp contrast to the ability to control one's eyes. Using one or both eyes without visual cues, it is difficult to generate more than a few circular rotations and it is even more difficult, for example, to control the radius of such movements without providing reference visual targets to track. Voluntary eye movements in the absence of looking at real or virtual visual cues are difficult and generally produce an uncomfortable feeling. Simply being told to “look away” without direction regarding where to look can easily create anxiety.
Thus, telling a device wearer to simply “look away” from an object without providing an alternate gaze point results in an action that can generally be performed, but is uncomfortable and not consistently repeatable. Looking away, for example, in a predominantly nearby (e.g., indoors) environment is likely to produce a very difference series of eye movements compared with looking away in a more expansive (e.g., outdoor) environment. Even when instructed to “look away” in a specific direction (e.g., left, right, up, down) and/or returning to viewing the original location, such eye movements are not consistent without visual cues. Unlike existing eye-tracking control devices, visual cues should ideally be at specific gaze locations in order to take advantage of physiological mechanisms such as memory-guided saccadic eye movements.
On the other hand, the eye and visual cortex are exquisitely designed to track real or virtual objects as they move about in different patterns. It is easy for most individuals to track a reference object (e.g., a ball or an image of a cursor) moving in a circular pattern. By following such visual references or cues (at gaze locations that are known to a device), it is possible to produce almost any pattern of movement using the eyes.
Along similar lines, a “swipe” or “sweep” motion of a finger or hand gesture is one gesture type used for machine control. If one attempts to “sweep” with one's eyes, unless eyes move in conjunction with real or virtual objects, one has little or no perception of any objects within the sweep pathway and one loses the ability to view what happened at the location where the sweep was initiated. It is also difficult to distinguish between an intentional eye sweep and a momentary glance or saccade to an object that might have attracted attention, for example, within the peripheral visual field of the device user.
Added to this, viewing of the eye is often obscured during normal function by eyelids and lashes. Furthermore, eye blinks in which the eyelid completely blocks viewing of the position of the eye must occur periodically for sustained function in order to maintain lubrication and the aqueous environment of the surface of the eye. Blink durations (normally lasting from 0.3 to 0.4 seconds) and velocities can be affected by fatigue, attentions, injury, medications, drugs, alcohol, and disease. Blinks can obscure an individual's vision and the viewing of the eye for up to 10% of the time.
Thus, new paradigms are required to discern intent from eye movements while retaining the ability of individuals to visualize and interact with their environment.